Here is what I would do. If I received a telephone call, I would ask to receive a written letter on the subject of meeting. Previously, I would have identified a local Lawyer who would work with me on matters such as this for a reasonable fee (they are around). I would give the letter to the Lawyer to answer. The answer that I would want is something like this: "Pursuant to your letter to Mr/Ms X dated XXXXX requesting a meeting, please be advised that I will be representing Mr/Ms X in this matter and he/she will not be in attendance. Unfortunately, I cannot meet with you on the day and time indicated in your letter. Please advise me of an alternate day and time that you would like to meet. I will appear at that time and we can pursue resolution of your questions. Please direct all future communications related to this subject to me via the contact information provided on this letter."
I am sure that the BOE will be instructed by the legal Dep't to drop the matter and not pursue. The BOE will also be afraid to say anything other than hello to you. At this point you are free not to return to the KH, or attend as you wish and not be bothered by the matter any longer as long as you keep your mouth shut and don't force their hand.
Nobody likes dealing with Lawyers/Law suits and the time and expense that goes along with it even if the Law suits are not real but just a possibility. The WTB&TS doesn't want to spend $$$ dealing with the legal cost of investigating and prosecuting personal indiscretions which do not threaten the congregation as a whole.